Real time video understanding applications utilize a machine learning classifier that processes video frames to determine the presence and/or properties of objects in a video image. For example, a security system may feed video frames to the machine learning classifier to determine potential threats. Other applications include self-driving cars and augmented reality applications.